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It is 7:30 A.M. on a cold November morning. Some fifteen people in various 
states of wakefulness have come together to discuss a county evaluation program. 
The evaluation staff is there; the evaluation advisory board representatives; the 
county board representatives; the state repres.^atatives ; and we represent the 
academic community. 

The evaluator explains what her staff has done during the year, the problems 
with getting started (fuzzy program goals, uncertain fundiilg), the data collection 
problems (lack of staff, little program cooperation, inconsistent state and county 
data processing- systems) , the management problems (unclear decision-making 
hierarchies, political undercurrents, trying to do too much), and the findings 
despite it all ("tentative to be sure but more than we knew a year ago and some 
solid recommendations") . 

Then the advisory board explains its frustration with the disappointing 
results of the evaluation program ("the data ju.st aren't solid enough") and the 
county board representatives explain why their decisions are contrary to evaluation 
recommendations ("we didn't really get the information we needed when we wanted it 
and it wasn't what we wanted when we got it"). The room is filled with disappoint- 
ment, frustration, defensiveness , cynicism, and more than a little anger. There 
are charges, counter-charges, budget threats, moments' of planning and longer moments 
of explaining away problems. Then the advisory board chairperson turns to us-- 
the neutral, academic observers, the evaluation experts who teach evaluation 
methodology at the Universi ty--and asks: "Tell us, what do we have to do to 
produce good evaluation research that actually gets used?" 

How many times has that question been asked in the last several years? 
There. .is no shortage of advice. At program evaluation planning meetings: "Make 
sure your program goals are clear and operationally defined." At county, city. 
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state board sessions and budget committees: ^'Specify clearly your information 
needs, make clear what ^information you really need to make decisions.*' At 
professional academic meetings: '*We need more rigorous methodologies, more 
sophisticated statistical analyses in evaluation research — and more funding' to 
conduct methodologically sound evaluation research/* At evaluation training sessions: 
**Make it clear to decision-makers what you can and can't do, be obj ective, maintain 
your integrity, work closely with relevant decision-makers, get your work done 
on time, and do the best job you can/* . 

' .The increasing demand for evaluation research has been matched by a prolif- 
eration of books on. how to do evaluation research (e.g., Weiss, 1972a and b; 
Suchman, 1967; Caro, 1971; American Institutes for Research, 1970; Rossi and 
Williams, 1972; Cherney, 1971, and Tripodi, Fellin, and Epstein, 1^71). Yet 
decision-makers continue to lament the disappointing results of ' evaluation research 

ye., ■ , . 

complaining that the findings don't tell them what they need to know. And 
evaluators continue to complain about many things, **but their most common complaint 
is that their findings are Ignored*' (Weiss, 1972a:319). 

THE ISSUE OF NGN-UTILIZATION 
. The problem of the non-utilization or the under-utilization of evaluation 
research has been discussed frequently in the evaluation literature. There 
seems to be a consensus that the impact of evaluative research on program decision- 
making has been less than substantial. Carol Weiss lists under-utilization as 
one of the foremost problems in evaluation research: 

Evaluation research is meant for immediate and direct use in improving 
the quality of social programming. Yet a review of evaluation experience 
suggests that evaluation results have not exerted significant influence 
on program decisions. (Weiss, 1972a: 10, 11 .) 

Other prominent reviewers have reached a similar conclusion. Ernest House 
(1972:412) put it this way: *Troducing data is one thing! Getting it used is -™ 
quite another/" Williams andLEyans (1969:453) write that "in the final' analysis. 
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the test of the effectiveness of outcome data is its impact on implemented policy. 
By this standard, there is a dearth of successful evaluation studies." Wholey 
(1971:46) concluded that '*the recent literature is unanimous in announcing the 
general failure of evaluation to affect decision-making in a significant way." 
He goes on to note that his own study "found the same absence of successful evaluations 
noted by other authors (1971:48) David Cohen (1975:19) finds that "there is 
little evidence to indicate that government planning offices have succeeded in 
linking social research and decision-making." Alkin (1974) found that Ti^,le VII 
evaluations were useful to project directors but were not useful at the Federal 
level because the results were not timely in terms of funding decisions. Weidman 
e^ al^ (1973:15) concluded that on those rare occasions when evaluation studies 
have been used, ". . . the little use that has occurred [has been] fortuitous 
rather than planned." 

The variety of advice for increasing utilization that we. found in the 
literature illustrates the complexity of the problem. It seems clear that no 
single prescription for success will suffice. The issue at this time is not 
the search for a single formula of utilization success, nor the generation of ever 
longer lists of possible factors affecting utilization. The task for the present 
is to identify and refine a few key variables that may make a major difference in 
a significant number of evaluation cases (cf. Weiss, 1972a: 325) . The research on 
utilization of evaluations described in this paper is a modest effort to_,|:ia,ve a 
bit further along that path of refinement. 

This paper is based on a. follow-up of twenty Federal health evaluations. 

'^This research was conducted as part of an N. I .M.H. -supported training program 
in evaluation methodology at the University of Minnesota. Trainees worked through 
the Minnesota Center for Social Research, University of Minnesota. The following 
trainees, in addition to the authors , participated in the project: James Cleary, 
Joan Dreyer, James Fitzsimmons, Steve Froman, Kathy Gilder, David Jones, Leah Harvey, / 
Gary Miller,- Gail Nordheim, Julia Nutter, Darla Sandhoffer, Jerome Seeal, and 
John Townsend. In addition, the following, Minnesota faculty made helpful comments on 
: an earlier draft of this paper: John Brandl , Director, School of Public Affairs; 

ERIC 5 . 
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We attempted to assess the degree to which these evaluations had been used and 



to identify the factors that affected varying deglrees of utilization. Given the 
pessimistic nature of most writings on utilization we began our study fully expecting 
our major problem would be to find even one evaruatibli that had had a significant 
impact on program decisions. What we found was considerably more complex and less 
dismal than our original impressions led us to expect. Evaluation research is 
used but not in the ways we had anticipated. Nforeover, we found that the factors 
we had expected would be important in explaining variations in utilization were 
less important than a new factor that emerged from our analysis. After reviewing 
our sample and methodology we shall report these findings and discuss their 
implications . 



The Sample 

The twenty case studies which constitute the sample in this paper are national 

health program evaluations. They were selected from among 170 evaluations on file 

2 

in the Office of Health Evaluation, HEW. In sampling these 170 evaluations we 
first eliminated studies v\rhich did not examine an existing program of some kind 
and studies which did not include some kind of systematic data collection so as to 
exclude policy pieces, think pieces, and armchair reflections from our analysis. 



(footnote 1 con^t.) 

Martha Burt, Tom.Dewar, and Ron GeiZer. Neala Yount transcribed over one hundred 
hours of interviews with unusual diligence and care. ^ 

2 

The Office of Health Evaluation 'coordinates most evaluation research in the 
health division of HEW. In 1971 this Office designed a new record-keeping system 
that collected abstracts of all evaluations coming through that office. 170 
evaluations were collected during the period 1971 --73. This becajiie the uriiverise of 
evaluations from which we chose our final sample. As part of this project a 
description and analysis of the content of all 170 abstracts were made. The 
results of that analysis can be found in Fatricia Grimes, Descriptive Analysis- of 
' 170 Health Evaluations , unpublished thesis. School of Public Affairs ,. University of • 
Minnesota, 1976. We wish to express our thanks to HEW officials for their assist- 
•ance throughout this research project, particularly Harry Cain, Director, Office 
of Policy Development and Wanning, Office of the Assistant Secretary for Health, 
and Isadore Seeman, Director, Office of Health Evaluation, Office of the Assistant , 
Secretary for Planning and Evaluation, DHEW. 

6. 
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We also eliminated studies that did not examine national programs and any studies, 
completed before 1971 or after 1973. We did this to enable a follow-up of 
evaluations which were recent enough to be remembered, and at the same time, 
evaluations which had been completed far enough in the past to allow time for 
potential utilization to occur. These control variables reduced the number of 
abstracts from 170 to 76 and gave us a more homogenous group of abstracts consisting 
of (1) program evaluation studies of (2) national scope where (3) some systematic 
data collection was done and (4) where the study was completed no earlier than 1971 
and no later than 1975. 

A stratified random sample of twenty studies was then drawn from among the 
"remaining seventy-six abstracts. Twenty was the number of studies we felt we could 
intensively , follow-up with our limited resources. The final sample was stratified 
by ''nature of program,'' as indicated in Table I, to guarantee a heterogeneous but 
representative group of health program evaluations. 



TABLE I 

Sample Stratification By' Nature of 
Program Evaluated 



Nature or Focus of the 
Activity which 
was Studied 

1. Training Program 

2. Program to Provide 
Education and General 

Information to Public 

3. Medical Treatment and 

Health Care Program 

4* Program of Basic Medical 
and Health Research and 
Basic Research in General 

5. Establishment, Imple- 
mentation and Planning* of ' 
Health-Related Programs 
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9. Other Programs 



TOTAL 



Frequency in 
Sample Frame 

17 

3 



29 



22 



■ 1 
76 



Percent in 
I Sample Frame 

22 .4 

3.9 

38.2 
5.3 

28.9 

1.3 . 
100.0 



Frequency 
in Sample 



5. 



__0 

20 ; 



For reasons of confidentiality neither the actual programs evaluated nor the 
titles of the evaluation studies can be reported* However, we can present a 
general description of the sar^ple. The twenty cases in this study consist of four 
evaluation3 of various Commuiaty Mental Health Centers program activities, four 
health training programs, two national assessments of laboratory proficiency, two 
evaluations of Neighborhood Health Center programs, studies of two health services 
delivery systems programs, a training program on alcoholism, a health regulatory 
program, a Federal loan forgiveness program, a training workshop evaluation, and 
two evaluations of specialized health facilities. 

Seventeen of the sample studies presented data in a way that allowed comparisons 
to be made between projects or activities. Sixteen included some type of quantitative- 
analysis and eight included some type of qualitative analysis. Nine of the studies 
included longitudinal data. Nineteen of the studies included outcome variables 
and fifteen included examination of implementation variables. Nine studies were 
based on data from non-random samples, three sampled randomly, and eight gathered 
data on an entire relevant population of projects or a unique project. Eighteen 
of the studies concluded with recommendations or explicit judgments while two only 
-presented data findings without making judginents. 

The types of evaluation studies in the final group of twenty cases range 
from a three week program review carried out by a single internal evaluator 
to a four year evaluation that cost a million and a half dollars. Six of the cases 
were internal evaluations and fourteen were external. 

Since it is impossible to specify the universe of evaluation research 
studies, it is not possible to specify the degree to which this sample of twenty 
cases is representative of evaluation research in general. The sample is diverse 
in its inclusion of a broad range of evaluations. We feel that this diversity 
and heterogeneity increases the meaningfulness of those patterns of utilization 
which actually emerged in our follow-up interviews because. those patterns were • 
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not systematically related to specific types of evaluations. The importance 
of this will become more evident when, after describing the nature of our data 
in the next section, we report our findings on utilization. 

Data on Utilization: The Interviews . 

—The first purpose of this study is to examine th<3 nature and degree of 
u tilization of Federal evaluation research. Ideally such a study would require 
exhaustive follow-up with any and all persons who did or could have used the 
study. With very limited resources it was- possible to interview only three key 
informants about the utilization of each of the twenty cases in the final sample. 
These key informants were (1) the project officer^ for the study, (2) the person 
identified by the project officer as being either the decision-maker for the 
program evaluated or the person most knowledgeable about the study's impact,^ 
' and (3) the evaluator who had major responsibility for the study. 

The project officer interviews were conducted primarily to identify informants, 
decision^makers , and evaluators who would be interviewed about the impact and 
utilization of the evaluations in our sample. (The results of those interviews 
will not be reported in this paper.) This snowball sampling technique resulted 
in considerable variation in who v/e interviewed as the ^'decision-makers'^ in each 
case. Most of these government informants had been or now are Office Directors 
(and Deputy Directors), Division Heads, or Bureau Chiefs . Overall , these 

^The term "project officer" refers to the person in the Federal government 
who was identified as having primary responsibility for administering the evaluation. 
For studies which were done by organizations which are not a part of the Federal 
government, the project officer was' the person who administered the Federal govern^ 
ment's contract with that organization. 

%e identified decision-makers by asking the project officers to name a 
person who would serve as an informant "about how the study was used in the 
government or elsewhere," a person who: 

"might be called a 'decision-maker' -vi2^a-yiz_ the study and its findings; * ~ 
"■"" who could -tell us what decisions, if any, ^ were , made on the basis of information 
. contained in the study." " ^ 
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decision-makers each represent an average of over fourteen years experience in 
the Federal government.^ 

The evaluators in our sample make for a rather heterogeneous group. Six of 
the twenty cases were internal evaluations so that the evaluators were Federal 
-administrators or researchers. Ir one case the evaluation was contracted from one 
unit of the Federal government to another so that these evaluators were also 
Federal researchers. The remaining thirteen evaluations were conducted by private 
organizations or non-government employees though several persons in this group'had 
either formerly worked in the Federal government or had since come to work directly 
in the government. Evaluators in our sample each represent an average of nearly 
fourteen 'years experience in conducting evaluative research.^,... 

Two forms of the interview were developed--one for government dceision- 
makers and one for evaluators. Each form was pilot-tested and revised before the 
final format was established. Both interviews are open-ended with questions 
covering the following areas: (1). Interviewee background, involvement in the 
program, and involvement in the evaluation; (2) purpose and objectives of the 
evaluation; (3) political context; (4) expectations during the stud/ about how 
the findings would be used; (5) major findings from the evaluation; (6) ways in 
which the study had an impact on program operations, program planning, program ^ 
policy, etc.; (7) non-program impacts, i.e.,, broader impacts on issues associated 
with the evaluation, position papers, new legislation, etc.; (8) impact and 
reception of specific study recommendations; (9) factors explaining the study's 
impact including specific questions on eleven factors (to be described later) taken 

^In two of our twenty cases we have no information on decision-maker exper- 
ience; this average is based on eighteen respondents. 

^ In four of our twenty cases we have no information on evaluator»s experience; 
..this average is based on sixteen respondents. 
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from the utilization literature on evaluation research; and (10) general questions 
on interviewee's observations about evaluation research and its utilization. 
The interviews were taped and transcribed, and ranged in length from one to six 
hours with an average of about two hours. 

The analysis of the interviews began with general discussions in which the. 
seventeen interviewers shared their perceptions about their own interviews. 
Three staff members then independently read all interviews looking for patterns 
and themes. These processes led to the formation o£ tentative hypotheses abotit 
dominant themes . The interview transcripts were then examined again, searching 
for evidence supporting these tentative hypotheses as well as looking for contra- 
dictory evidence and counter-examples. Quotes extracted from the interviews as 
examples of particular points were then independently examined by other staff 
members to check for context and accuracy. Only those findings about which th^re 
was a high degree of consensus are reported here. 

IMPACT OF EVALUATION RESEARCH 

The conceptualization and operationalization of the notion of research impact 
or evaluation utilization is no easy task.. We began with an ideal-typical construct 
of utilization as immediate and concrete effect on specific decisions and program 
activities resulting directly from evaluative research findings. Yet, as noted 
earlier, the consensus in the evaluation literature is that instances of such 
impact are relatively rare. 

Given the dismal conclusions of most studies of utilization we began our 
study anticipating that our major problem would:be finding even one evaluation 
that had had a significant and identifiable impact on program decisions. Because 

'^This paper represents the initial and general results of bur analysis, 
A more extensive and detailed description of the sample, methodology, and analysis 
is presented in Nancy J. Brennan, Variation in the Utilization of Evaluation Research 
in Decision Making , unpublished Ph.D. dissertation, University of Minnesota, forthcoming. 
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we expectei little evidence of impact and because of our inability to agree on 

an operational definition of utilization, we..adopted an open-ended strategy in 

our interviewing which allowed respondents to define' utilization in terms meaningful 

to them. Our question was as follows: 

Now weM like-. to^fociis on the actual impact of this evaluation study. 
We'd like to get at any ways in which the study may have had an impact — 
an impact on program operations, on planning, on funding, on policy, on 
decisions, on thinking about the program, and so forth. 

From your point of view, what was the impact of this evaluation study on 
the program .we *ve been discussing? 

Following a set of probes and additional questions, depending upon the 

respondents* initial answers, we asked a question about the non-program impacts 

. .. of ^ thje evaluation . . . . ^ ^ j _ _ 

We've been focusing mainly on the study's impact on.^the program itself. 
Sometimes studies have a broader impact on things beyond an immediate 
program, things like general thinking on issues that arise from a study , 
or position papers , or legislation . . . 

Did this evaluation have an impact on any of these kinds of things? 

What we found in response to these questions on impact was considerably 
more complex and less dismal than our original thinking had led us to expect. 
We found that evaluation research is used by decision-makers but not in the 
clear-cut and organization-shaking ways that social scientists sometimes believe 
research should be used. The problem we have come to feel may. well lie more in 
many social scientists' overly grand expectations about their own importance to . 
policy decisions than in the intrarisience of Federal, bureaucrats. The results of 
V our interviews suggest that what is typically characterized as undercut ilization ^ 
or non-utilization of evaluation research can be attributed in substantial degree 
to a.definition of utilization that is too narrow and fails to take into consid- 
eration the nature of actual decision-making processes in most programs. 



-11- 



The Findings on Impact 

In response to the first question on ir'^ fourteen of eighteen responding 
decision-makers and thirteen of fourteen respondir;g evaluaiors felt that the 
evaluation had had an impact on the program, (Two of the fU^ : rs and 

six of the evaluators felt that they had too little dir *c actual 

use to comment.) Moreover, thirteen of sixteen responding ^^wwiiaJLon-makers and 
nine of thirteen responding evaluators felt these specific evaluation studies had 
had identifiable non-program impacts, .. . 

^ The number of positive responses to the questions on impact are quite striking 
^considering the- predominance of the theme of non- utilization in the evaluation 

literatufe. Tlie'main difference here, however, may be that the actual participants 

in each specific evaluation process were asked to define impact in terms that were 

meaningful to them and their situations. Thus, none of the impacts described 

was of the type where new findings from an evaluation led directly and immediately 

to the making of major, concrete program decisions. The more typical impact was 

one where the evaluation findings provided additional pieces of information in 

the difficult puzzle of program action permitting some reduction in the uncertainty 

within which any Federal decision-maker inevitably operates. 

The most dramatic example of utilization reported in our sample was the 
case of an evaluation of a pilot program. The program administrator had been 
favorable to the program in principl e, . was uncertain what the results would be, 
but was ^'hoping the results would be positive.** The evaluation proved to be 
negative. The administrator was '^surprised, but not alarmingly so. . , We had 
expected a more positive finding or we would not have engaged in the pilot studies." 
(DM367 :.13)^ The program was subsequently ended with the evaluation carrying 
'♦about a third of the weight of the total decision.*' (DM367:8) 

^Citations for quotes taken from the interview transcripts will use the 
following - formatv (DM 367:13)- refers ^to the transcript.^of- an 4nterview..^ — 

o 13 
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This relatively dramatic impact stood out as a clear exception to the more 

typical pattern where evaluation findings constitute an additional input into 

an on-going, evolutionary process of program action. One decision-maker with 

twenty-nine years experience in the Federal government, much of that time directing 

research, gave the following report on the impact of the evaluation s' - about 

which he was interviewed: 

It served two purposes. One is that it resolved a lot of doubts and 
confusions, and misunderstandings that the advisory conunitte had, , . 
And the second one was that it gave me additional knowledge to support 
facts that I already knew, and, as r say, broadened the scope more than 
I realized. In^bther words, the perceptions of where the organization 
was going and what it was accomplishing were a lot worse than I had 
anticipated. . ., but I was somewhat startled to find out that they were 
worse yet it wasn * t very hard be partly ^ c^^ ^ 

that I was observing. (DM232:17) 

He goes on to say that following the evaluation, . , 

we changed our whole functional approach to looking at the identification of 
what we should be working on. But again I have a hard time because these 
things, none of these things occurred overnight, and in an evolutionary 
process it^s hard to say, you know, at what point it made a significant 
difference or what point did it merely verify and strengthen the resolve 
that you al ready had , (DM232:17) 

This decision-maker had become highly involved in applied government 

research, including his initiation of the study in our sample, because he believed 

research can help reduce uncertainty in decision-making. 

As time came on I more clearly recognized that I was not satisfied with 
• having to make program decisions that I was making or that others 
. were making based on ^'professional judgment." Not that it's bad or anything. 



(footnote 8 con't.) 

a decision-maker about evaluation study number 367. The quote is taken from 
p. 13 of the transcript. The study numbers and page numbers have been 
systematically altered to protect the" confidentiality of the interviewees. 
The study numbers do not correspond to any codes used with in DHEW. Thus 
(EV201:10) and (PO201:6) refer to interviews about the same study, the former 
was an interview with the evaluator% the latter was an interview with the 
project officer. ' \ 

14 
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it's just that it's pretty shaky at times, and you know, you always sit 
back and say,^*now if I hadn't done that and done something else, what 
would have been the result?** So it's nice to find that there are better 
ways of doing it. (.DM232:25) 

Still his 'assessment of -the actual impact of the evaluation was quite constrained: 

"It filled in the gaps and pieces that various ones really had in their orientation 

to the program. (DM232:12) "It verified my suspicions." (DM232:24). 

Respondents frequently had difficulty asses^^ :he degree to which an 

evaluation study actually affected decisions . u: Je af completion of the 

evaluation. This was true, for example, in the case of a large-scale evalu- 

ation effort that had been extremely expensive and had taken place over several 

years time. The evaluation found some deficiencies in the program, but the, 

overall findings were quite positive. Changes corresponding to those recommended 

in the study occurred when the report was published, but those changes could not^ 

be directly and si;;tply attributed to the evaluation: 

The staff was aware that the activities- in the centers were deficient 
from other studies that we had done, and they were beefing up these budgets 
and providing technical assistance to some of the projects and improving 
mental health activities. Now I can't link this finding and that activity. 
• Again that confirms that finding and you say, eureka, I have found 

^ deficient, therefore I will [change] the program... That didn't 

happen. [The] deficiency was previously noted. -A lot of. studies like this 
confirmed what close-by people know and they were already taking actions 
before the findings. So you can't link the finding to the action, that's just 
* confirmation. . . The direct link between the finding and the program decision 

' is very diffuse . [Its major impact was] confirming our setting, a credibility, 
a tone of additional credibility to the program. (DM361:1,2,13) 

Moreover, this decision-maker felt that additional credibility for the 

program became -one part of an overall process of information flow that helped 

to some degree reduce the uncertainty faced by decision-makers responsible 

for the program. "People in the budget channels at 0MB were, I guess, eager 

for and interested in any data that would help them n)ake decisions, and this 

was certainly one usefu' bit of data." (DM361:13) 

The kind of impact we found, then, was that evaluation research provides 

some additional information that is judged .and used in the .cont^jct.of..otber _ _ 
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available information to help reduce the unknowns in the making of difficult 

decisions* The impact ranges from *'it sort of confirmed our impressions. / 

"donfirming some other anecdotal or impression that we had'* (DM209:7,1) to 

providing a new awareness that can carry over into other programs: 

Some of our subsequent decisions on some of ..our other programs v;ere 

probably based on information that came out of this study. . , The 

most significant information from this study that we really had not 
realized. . .made an impact on future decisions with regard to 
other programs that we carry on. fnM209;'' 

And why did it have this iii5)act? 

Well I guess T Ml go back to the points I've already made, that it 
~ confirmed" some impressionistic feelings and anecdotal information that 
we had"^bout certain kinds of things. At least it gave us some hard data 
on which to base some fiiture programming decisions. It may not have > 
been the only data, but it was conHrm that '^s " 

important. . . And you know at the time this study was conceived, and 
even by the time it was reported to us, we really had very little data, 
and you know, probably when you don't have any data, every little bit 
helps . (DM209; 15) 

This reduction of uncertainty emerged as highly important to decision- 
makers. In some cases it simply ma di^ them more confident and determined. 
On the other hand, where thcs need fcx change is indicated an evaluation stu^ 
can help speed up the process of chsags or provide an impetus for finally 
getting things rolling. 

Well I think that, all we did was probably speed up the process. I 
think that they were getting there anyhow. They knew that their 
performance was being criticized by various parts of the government : 
and the private sector. As I said earlier, we didn't enter this study 
thinking that we were going to break any new ground, and when we got 
finished, we knew that we hadn't. *A11 we did was document what the pecple 

have been saying for a long time—that aire doing a roiasy job, 

so what else is new? But we were able to show jusi: how poor a job they 
were doing, (EV268:12) 

f^^d^^icing uncertainty, speedii^ :;>xings up, and gettimg things finally started 
are x^al impacts--not revolutionary. CTganizacion-shaking impacts--but real, 
important impacts in the opinion of ix&e people we interviewed. One administrator 
summarized this view both on the specific evaluation in question and about 
" evaluation in general as follows : g 
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Well, I've worn several hats. I've been on evaluation teams. I've 
participated in extensive evaluation in-house of other organizational components. 
Myself I have a favorable view toward evaluating. If nothing else it precip- 
itates activity many times that could not be precipitated without someone 
taking a hard look at an organization. It did precipitate activity in 
[this program]. Some of it was not positive. Some of it was negative. At 
least something- occurred that wouldn't have occurred i f the eva lu ation hadn 't 
taken place . (DM312:21) ~ ■ \ 

Another evaluator made it quite clear that simply reducing the enormous uncertainty- 
facing many program administrators is a major purpose of evaluative research. 

One of the things I think often is that the governnient itself gets ! 

scared. . .of whatever kinds of. new venture that they want to go into, and . . r^- 
they're quite uncertain as to what steps th^y want to take next. So then 

they say, okay, let's have some outside person do this for ..usV or maybe^^^^ ^^"^ "^^^^ 

an inside person do this, so at least we have some "data" to base- some of "^^^^^ • 

, pur policies on. (EV283:34) \. ', ; , 

The view of evaluation research that emiefges in bur inter ''^ ™n 

stark contrast to the image of utilization that is presenJ^t as the ideal in the 
bulk of the evaluation literature, or i^ast Jre impression with which that 
literature left us. The ideal held forth; im Tthe literature we reviewed earlier 
is one of major impact on concr^jte decis2Mris.« Trie image that emerges in our inter- 
views is that there are few. major, di recti on-^aaauayging decisions in most programming^ , " 
;and that evaluation research is used as one pfrS^ce of information that feeds into 
a slow, evolutionary .pirocess of program -fjwelopment. Program development is a 
process of "muddling through" (Lindblom„ 1^1$^; Allison, 11971; Steinbruner, 1974) ; ■ 

and evaluation research is part of the m^tdd'^^ng. 

Neither did we find much expectatioi^ fh$$t government decision-making could 
be or should be otherwi^ia. One person Wi th thirty-five years experience utrt the 
Federal government (twenty of those years xtij eva.lua^t:\on) put It like this:: "I 
don't think an evaluation's ever totally used, that was true whether I was using 
them as an administrator or doing them-myseljf , /-^ (EV346;1I) Later in the interview 
he said: 

I don't think the government should go out and use every evaluation it gets. 
I think sometimes just the insights of tti^ evaluation feed; over to the 
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next administrative reiteration, maybe just the right way to do it. That 
is, [decisions aren * t] clearly the result of evaluation. There*s a 
feedback in some way, , upgrading or a shifting of direction because 
of it. [Change]' it is, you know, small and slow. . . (EV346:16) 

An evaluator expressed a similar view. 

I think i.t*s just like everything else in life; if, you're at the right 
place at the right time, it can be useful, but. it's obviously only probably 
one ingredient in the information process. It's rather naive and presumptuous ; 
on the part of the evaluation community and also it presumes a ratibnality 
that in no way fits. (EV264:i8).. . 

Our findings, . then, suggest that the predominant image of non~utilization 
that characterizes mux:h" of the commentary on evaluation research can be attributed 
; in substantial degree to a definition of -utilization that is too narrow in its . ; 
emphasis on seeing immediate, direct, and concrete impact on program decisions. 
Such a narrow definition fails to take into account the nature of most actual 
program development processes. 

Perhaps many social scientists have come into applied government research 
with high hopes of rationalizing the system. Like the Peace Corps volunteers 
of the sixties who set off to change the world and ended up touching a few 
villages and a few individuals, many evaluators seem to have entered the arena 
of applied research expecting to make great policy waves and are disillusioned 
to find that they've only provided a few cogs in the great gears of program 
change and development, helpedwith a decision here or there, made actions more 
certain for a few decision-makers. \ ' ' 

Yet the situation seems little different in basic research. Researchers 
in any field of specialization can count the studies of major impact on one 
hand. Most science falls into that great amorphous activity called **normal 
science ' Changes come slowly. Individual researchers conrlbute a bit here 
and a bit there, reducing uncertainty gradually over time. Scientific revolutions 
are infrequent and slow in coming (Kuhn, 1972). ■ / 
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The situation is the same in applied research. Evaluation research is one 
part of the normal ''science^^ of government decision-making. Research impacts in 
ripples not in waves. Occasionally a major study emerges with great impact. 
But most applied research can be expected to make no more than a small and momentary 
splash in the great pond of government. The epitaph for most studies will read 
something like this: 

[We expected that, it would be used] but in a way of providing background 
information around the consequences of certain kinds of Federal decision- 
making options. But not necessarily in and of itself determining those 
decisions. In other words you might have some idea of what the consequences 
of the decision are, but there might be a lot of other factors you'd take 
into account iim how you would de:cide; . . CDM264:8) ^ ^ " " 

You know, impact in some of these things implies "that this one 
thing is going^ to affect things, [but] it Vsvpart of a total atmosphere, 
and in the balance of things atis contributing another bit of information 
about the importance of this particular process, but by no means is it 
the only thing entering into what's going on in a policy review like that 
at that- time. 



[It had a particular impact in that] it contributed to. the general 
information context of what was going on at the time, rather thah in itself. . 
It contributes to thai: background of understamding oae of the policy issues, 
rather than resulting in one option versus another of policy being adopted. 
(DM264:11) / 

FACTORS AT^EECTING UTILIZATION 
We began this paper with the observation that there is .jio shortage of advice, 
about how to increase the utilization of evaluation research. At the same tim^, 
it is clear that no single prescription for success will suffice. A fairly sub- 
stantial list of ' variables or factors that may affect the degree of research impact 
has been generated by the literature on utilization of social. science in decision- 
making. These explanatory factors fall into three basic catEggories: 

1. Characteristics of tfee organization 

a. The constraints 33f decision-making in national -bureaucratic organ- 
izations C6-g.> li^mpson, 1967) 

b. New and innovati-yxL agencies versus older, established agencies 
(Weiss, 1972a) 
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Communication patterns in organizations (Mitchell, 1973; Jain, 1970.) 
Level in the organization where evaluation is done (Alkin, . 1974) . 
The degree o£ politicization of the organization (Weiss, 1972b). 



2. Characteristics of Actors in the System: Evaluators and Decision-Makers. 

a. Two -Communities Theory, different values, languages, reward systems, 
and affiliations (Caplan et^ al^^, 1975; Halpert, 19^9; Engstrom, 1970) .. 

b. Academic versus private contract evaluators: (Bernstein and Freeman, 1975) 
'c. Experience in doing and using research (Clock, 1961) . 

d. Leadership (Engstrom, 1970) .J 

e. Internal versus external evaluators (Weiss , 1972a) , ' 
Repmttion an4:legi.tdmacy of the evaluator (Archibald, 1970) . 

3. Characteristics of the Evaluation 

a. Methodological Quaillity (^emstein and Freeman, 1975; Weiss, i972a; 
-Cohenv 1975; Glarex and-3^^^ — - - ; r - 

b. Relationship to original objectives (Glazier., and Taylor , 1969) . 

c. Biass (Mitchell, 1973) • • ■ 
d: Compi^exity CGldck^^^ " 

e. Degree of dissemination (Halpert, 1969). 

f. Relevance to agency. 

g. format of the report (Alkin, 1974: Glazer and Xaylorvv'1969),. . * . 

h. Positive versus negatirve findings (Weiss, 1972a) • 
. i. Timeliness (Mitchell, 1973)/ - 

j. Formative versus summative (Scriyen, 1967; Alkin, 1974). 

k. Scope of recommendatiocns for change (Weiss, 1972a). 

This list of tpossible factors ±s only meant to be saggestive not exhaustive. 

It is unlikely that any single study on utilization wilU be able to examine the 

importance of all such factors. The task for the present is to identify and refine 

a few key variables that may make a major difference in a significant number of 

evaluation cases (of. Weiss, 1972a:325). 

ThelData on Factors Affecting .Utilization 

In our own research we "took S dual approach to' this problem of variable 

spe:cifi cation. Onre the respondents had.~dxscussed their perceptions about the nature' 

and degree of utiJiLzation of the specific evaluation study under investigation we.^ 

aske,d the following open-ended question:: 

Okay, you've described the impact oS the study. Now we'd like you to think 
about why this study was used in the ways you've J^ust described. Some of this 
you 've"'l[lr^ady done, but now we'd like to eXpilorfe ^Ife in more detail. 
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What do you feel were the important reasons why this, study had the level of 
impact it did? 

(CLARIFICATION, IF NECESSARY :) 

The literature on evaluation studies suggests a lot of reasons why some 
evaluations are used while others. ..are ignored. Most of the literature, 
however, is based on speculation. A major objective of the interviews 
doing is to find out directly from people who are in a position 
uu know, '^.hat factors they consider important in explaining how specific 
studies are used. 

following a set of probes and follow-up questions, depending on tfe initial 
response to the above question, we asked respondents to comment on .the irrelevance and 
importance of el. even factors extracted from the literature on utili~2atxDn: 
methodological quality, methodological appropriateness, timeliness , lateness of 
report., positive/negative findings, surprise of findings, central/ peripheral • 
program objectii?T^?s evaluated, presence/ absence of related studies, political factors, 
goT^ernment-evaltotor interactions, and resources available for the study. Finally, 
we .asked responiti.ents to ''pick out the single factor you feel had the greatest 
effect on how this study was used." ' ^ ■ 

Two related factors emerged as important in our interviews: (1) a political 
considerations factor and t^) a factor we have called the personal factor. This • 
latter factor was ^unexpected and its clear importance to our respondents has, we 
"rbelieve, substantial implications for the utilization of evaluaition research. None 
of the other specific literature factors about which we asked- questions emen, ed:^s^ 
important with any- consistency. Moreover, when these specific factors were important 
in explaining the utilization or non-utilization of a particular study it was virtually..:^ 
always in the context of a larger set of circumstances and conditions related to the 
issues and decisions at hand. 

y In the pages which follow we shall briefly review our findings with'regaroi to 
the importance of these specific factors and then examine in some detail the major 
factor which did emerge as consistently important to an understanding of vari'atiions ' 
in utilization :Qf evaluation research. 
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Lateness of Study Completion 

There is a general impression from th^ 3rature cited earlier thai much 
evaluation research is under-utilized because studies are completed tuo late to 
be used for making a specific decision, particular ly_.-budgetary decisions. This 
problem is based to a large extent on the notion that the purpose of evaluation 
research is to serve as the basis for the making of specific, identifiable, and ^ 

: concrete decisions . Inasmuch as we have already argued that most evaluation research 
does not serve such a narrow function and is not intended to . serve such a narrow 
function, it is not surprising that lateness in the ^completion of studies was not 

: an important factor in explaining utilization of the studies in our sample. 

" ^ In four -of our twenty cases deci si on-- malcersiridi^ the final' r¥sea^^ 

reports were completed late, but in all four cases preliminary information was 

available to a sufficient extent to be used at the time the study should have been 

completed. In no case was lateness considered the critical factor in explaining 

the limited utilization of the studies. Rather, the information was viewed as 

feeding into a longer term process of program development and .decision-making. 

Several decision-makers commented that it was helpful, to have, the information on time, 

but had the final report been late the impact of the study would not likely have 

been different. This is partly because few issues become . one- time decisions. As 

one decision-maker put it; 

[The] study was too late for the immediate budget: that it was siiggiosed to 
impact on, but it wasn't too late in terms of tite Tact that the same issue 
was occuring every year after that anyway. ;(DH264:16) 

Another decision-maker made it quite clear that the evaluation was aimed at 

•a broader impact than the meeting of a specific deadline. 

No, this had no time.table on it as far as I know. That we had rto geit this done 
by such and such a time in order to make such a decision. I think it was part 
icrf a grand, overall plan, rather than as something specific in oirder to do 
something. I think it Was in a bigger thing than for any ona:^derisiQn . 
(DM366:13) 
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Methodological Quality and Appropriateness 

The major factor most often identified as the reason for non-utilization 
is t^ of much evaluation research. Of the fifteen decision-makers 

who rated methodological .quality of the study about which they were interviewed, 
five rated the methodological quality as '^high/' eight said it was ''medium,'* 
and only two gave the strady a '*low'' rating. Of seventeen responding evaluators 
there were seven "high'Vxatings, six "medium'' responses, and four "low" ratings. 
No decis ion- maker and OEEly. one evaluator felt that the methodology used was 
inappropriate for researching the question at issue. 

More to the pointy only four decision-makers felt that methodological Equality 
was ."very important" in .explaining t J V ~ 

however, revealed that "methodological quality" meant different things to different 
decision-makers. For some it meant the reputation of the evaluators; for others 
it meant asking the right question. In no case was methodological quality ^identified 
as the most important factor explaining either utilization or non-utilization . . • 

The relevance of methodological quality must, be uriders^ full 

context, of a study, the political environment, the degree of uncertainty with which 

the decision-maker is faced and thus his/her relative need for ; ariy^^ ^^^^^ 

clarifying information. If information is scarce, then new information of even 

dubious quality may be setnewhat helpful. For example, one administrator admitted 

that the evaluation's metihodological rigor could be seriously questioned, but the 

study was highly useful In policy discussions* 

The quality and: the methodology were not even considered. All that \was 
considered was that management didn^^t know what was going on, the terms, 
V ! the procedures, the program was foreign to their background. And they did 
jiot have expertise im it, so they were relying on somebody else who had the 
. expertise to translate to them what was going on in terms that they would 
understand and what :t3ie ^nroblems were . (DM312: 17) - 

Social scientists may lament this situation and may well feel that the 

methodology of evaluation Ji^esearch ought to be of high quality for value reasons. 
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i.e., because poor quality studies ought not be used . But there is little in 
our data to suggest that improving methodological quality in and of itself will 
have much effect on increasing the utilization of evaluation research . No matter 
how rigorous the methodology and no matter how sophisticated the statistical 
. manipulations, evaluation research iiri-most instances will still be only one piece 
. of information in a complex and evolutionary process of program decision-making 
and development. 

Again, the importance of methodological quality as a factor explaining util- 
ization is tempered by the nature of the utilization we found. Were evaluations 
being used as the major piece of information in making critical one-time decisions, 

met ho do logical rigor might be paramount", But where evaluatibri'r 

part, often a small part, in a larger whole, decision-makers displayed less than 
burning interest in methodological quality. Indeed, methodology was most likely 
to be called into question if the evaluation was expected to play a central role in 
the making of a decision or if the results were particularly negative or surprising. 

- One highly experienced administrator was quite explicit about this from a program 
point of view. 

Well, let me put it in another context. If it were negative findings 
programmatically we would have hit very hard on the methodology-and tried 
to discredit it. You know, from the program standpoint. But since it was 
kind of positive findings, we said, 'Okay, here it is.* I£ anybody asked 
" us about the methodological deficiencies we were never reluctant to tell them 
what we thought they were. Not many people asked . (DM361 ; 1 5) 

Political Factors " 

This last quote on methodological quality makes it clear that methodology, 

like everything else in evaluation research, can become partly a political 
' question. The political nature of evaluation research has been well -documented. 

The dejcision-^makers and evaluators in our sample demonstrated an acute awareness 
; of the fact that social science research rarely produces clear-cut findings. 

Findings must be interpreted and interpretation is partly a political process, 
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a value^laden process where truth is partially a matter of whose ox is being gored. 

Of the eleven specific factors about which we asked respondents to comment, 
political considerations were most often discussed as an important factor' in 
explaining how study findings were used. Nine decision-makers and ten evaluators 
said that political considerations had affected how the study was used. In combin- 
ation, at least one person interviewed in fifteen of the twenty cases felt that ■ — 
politics had entered into the utilization process^ Nine decision-makers and seven 
evaluators felt that political considerations had been ''very important'^ as a factor 
explaining utilization. On the other hand, nine decision-makers and five evaluators 
reported that political considerations played no part in the utilization process . 

There is not space here to fully explore the nature and impact of these 
political factors. They include intra-agency and inter-agency rivalries; budgetary 
fights with 0MB, the Administration, and Congress; power struggles between Washington 
administrators and local program personnel; internal debates about the purpose and/or 
accomplishments of pet programs. Budgetary battles seemed to be the most . 

political. One evaluator was particularly adamant about the political nature of 
his evaluation from the initiation of the study to the final report: 'This was 
a really hot political issue, and I think the political aspects were developing 
and changing, and I think that was the really important factor^^ 
utilization]." (EV264:173 The decision-maker concurred. 

We did not find, however, that political factors suddenly and unexpectedly 
surfaced once a study was completed. In almost every case both the decision- 
makers and evaluators were well aware of the political context at the outset. 
Moreover, our respondents seemed to feel political awareness on the part of everyone 
involved was the best one could expect. Social scientists will not change the 
political nature of the world, and while several respondents were quite cynical 
on this point, the more predominant view seemed to be that government would not 
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be government without politics. One particularly articulate decision-maker 

expressed this view quite explicitly; 

This is not a cynical statement. * . A substantial number of people 
have 2Ln improper concept of how politics works and what its mission is. 
And its mission is not to make logical decisions, unfortunately for those 
of us--who think program considerations are important . . Its mission is to 
detect the will of the governed group and express that will in some type 
of legislation or government action. And that will is very rarely^ when- : 
' it's pooled nationally, a rational will. ' It will have moral and ethical 
overtones, or have all kinds of ein..>tional loads. . . 

It's not rational in the sense that a good scientific study would 
allow you to sit down and plan everybody ' s 11 :^e; and I *m glad it ' s not , 
by the way. Because I would be very tired very early of something that 
ran only by the numbers . Somebody 'd forget part - of the numbers,^. s 
not fighting the system, but I aa saying that you have to: be^^^^^^^ ' 
what you expect from sl rational study when you insert it into the system . , 
It has a tremendous impact. . . It is a political, not a rational process. * 
Life is not a very simple thing . (DM328: 18-19) 

The importance of political considerations in much (though clearly not all) 
evaluation research can be partly understood in terms of our emphasis on the , 
' role of evaluation in reducing uncertainty for decision-makers. Several organ- 
izational theorists (e.g. Thompson, 1967; Crozier, 1964) have come to view power 
and relationships within and between organizations as a matter of gaining control 
through the reduction of uncertainty. French sociologist Michel Crozier has 
summarized this view as follows: 

.In such a context, the power of A over B depends on A''s ability to predict 
B^s behavior and on the tincertainty of B about A' s behavior. As^long as 
the requirements of action create situations of uncertainty, the individuals, 
who have to face them have power over those whol_iare affected by the results 
of their choice (1964:158). 

More directly, James Thompson (1967) describes evaluation research as one ' 
major organizational mechanism for reducing internal as well as environmental 
uncertainty. He argues that the methodological design of much evaluation research 
can be predicted directly from the political^ function that assessment plays. We 
believe that our data directly supports this viewpoint. Evaluations are undertaken 
as a mechanism for helping decision-makers cope^ with the complexity of the programs 
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■for which they have responsibility. As one weapon or tool in the struggle to gain 
control over organizational and program processes evaluation research can fully 
be expected to take on a political character. Indeed, as Thompson argues, it is 

; completely rational for decision-maJ-^rs to use evaluations in a political fashion 
for control and reduction of uncertainty. 

It -would appear to us that it behooves social scientists to Inform themselves 

: fully^bout the political context of '^the evaluations oh which they work. It is 
precisely through such a heightened awareness of the political implications and 

. consequences of their research that social scientists can reduce their own 
uncertainty about the uses to which their work is put without impairing their 
ability to state their '^truth'* as they see it . 
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Other Factors Affecting Utilization 



None of the other factors about which we asked specific questions emerged 



as consistently important in explaining utilization. When these other factors 



were important their importance stemmed directly from the particular circum- 



stances surrounding that evaluation and its purpose, particularly its political 



purpose. . For example, the amount of resources devoted to a study might add to 



the credibility and clout of a study but more costly evaluations did not show 



any discernible patterns of utilization different from less costly evaluation. 



The resources available for the study were judged inadequate for the task at ^ 



hand by only two decision-makers and five evaluators. 



Whetheror not findings were positive or negative had no demonstrable effect 



on utilization. We had studies in our sample in which the findings were rated by 



respondents as predominantly negative; other studies were predominantly positive 
in their conclusions; and still others had mixed findings. This variation was 
evenly distributed in our sample. Interestingly enough, the decision-maker and 
evaluator on the same study often differed on whether findings weze "positive'' 
or "negative,*' but despite such disagreements neither rated the positive or 
ne gative nature of the findings as p articularl y im portant in e x plainin g eit her 
utilization or non-utilization of the evaluation. 

The unimportance of this factor in explaining variations in utilization is 
related to the fact that, as*\.noted earlier, the positive or negative findings of a 
particular study constitute only one piece of information that feeds into a larger pro- 
cess in such a way that no single study is likely to have a dramatic impact on a 
program. Negative or positive findings are thus interpreted in the larger context 
of other available information. 

Furthermore, the negative or positive nature of an evaluation report was 
unimportant as a factor explaining utilization because such findings, in either 
direction, were virtually never surprising. Only four decision-makers expressed 
^ciixprise at the findings of the study. Only one decision-maker felt this surprise 
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had an important effect on utilization. There was considerable consensus that 

surprises are not well received. Surprises are more likely to increase uncertainty 

rather than reduce uncertainty. 

One decision-maker took this notion a step farther and made the point that a "good" 

evaluation process should build-in feedback mechanisms that guarantee the relative 

predictability of the content of the final report. 

If you're a good evaluator you don't, want suiTprises. The last 
thing in the world you want to do is surprise people, because the ... 
chances are surprises are not going to be well received ... It isn't 
a birthday party, and people _ar^n't i-eally looking for surprises . So 
that if you're coming up with data that is different than the conven- 
tional wisdom, you ought, c good evaluation effort I would suggest, would 
get those ideas floated during the evaluation process so that when the 
final report comes out, they aren't a surprise. So my reaction was 
that if you were dealing in the world of surprises you aren't doing a 
very good evaluation. Now you could cor up with findings contrary to 
the conventional wisdom, but you ought uo be sharing those. _idea?;, if 
you will, with the people being evaluated during the evaluation process 
to be sure that those surprises don't have any relationshipto reality and 
again working during that process on the acceptance, that maybe. .. , but if 
you present a surprise, it will tend to get rejected. See, we don't 
want surprises. We don't like surprises around here. (DM346 : 30-31 ) 

Th3 evaluator for this project expressed the same opinion: "Good 
managers are rarely surprised by the findings. If there's a surprising finding 
it should be. rare. I mean, everybody's missed this insight except this great 
evaluator? Nonsense!" (EV364:13) 

Surprises were occasionally helpful if they related tic relatively minor 
aspects of , the program where fine-tuning could take care of. the problem. But 
there was a marked emphasis throughout the interviews on the importance of eval- 
uation as a way of "reinforcing" already existing information, adding "credibility" 
and confirmation to existing or anticipated program directions, and essentially 
making the already suspected more certain. 

Surprises may have been especially minimal in our sample because most of 
the studies examined central rather than minor or peripheral program objectives. 
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It is sometimes suggested that evaluarions aren't used because they concentrate on 
minor: 11 jsues. But only two decisisJib^BSkers and one evaltiator felt that the evaluation 
in qac&^^fron de?lt with peripheral prxzggram objectives. On tl^ other hand niii^ 
decisis 'iVte'-makers an^' eight evaluatSES ^eliz that a mar->f^ ^ac^^^^s^ in utfJLization vafi 
#^etl[iear Q>t not tte evaluation e3cami*S5^?(H i^^'-tral program ohf^m^/es. The most useful 

^it4tW^ were those that focaisieu m central objectives Imat these were also 
prec5 : e . the kinds of evaluaticms ^iV would not produce information that ii: ^ fiiid 
of i^t^it, f could change a major poli^vr -:^irection. This may have been because 
focu?5l^ , on major objectives the^liid-f^s in our sample bassame one part in thf 
larger policy process while at the sssre time reducing the potential for the more 
immediate, concrete impact that a study might nave had if it had been aimed at 
some peripheral, easily-changed policy objective or program component . We 
lacked sufficient cases of the latter type, however, to explore this possibility 
more fully- 

Another factor of interest to us concerned the point in the life of the program 

when the evaluation Jtook p]ace. The interview question read as follows: 

The next factor concerns the point in the life of a 
program when an evaluation takes place. For example, some 

. e valuatipns are done so earl y in the life of a program that 

the program is still changing a great deal and so the evaluation 
is already out-of-date by the time it is finished. Other 
evaluations are done so late in the life of a program that the 
decision to terminate has already been made by the time the 

evaluation is done. These examples represent the extremes, but - ■ 

at what point in life of your program did this evaluation take place? 

Our sample contained studies that were done at all stages in the lives of 
programs. The key point that emerged with regard to this factor was that different 
■■-. questions emerge at different points in the life of a program. Early in the pro- 
gram the most useful information concerns procedures and implementation. ..Outcomes 
only become important after the program has been operating for a resonable period. 
Budget and cost issues become central late in the program»s life. Our respondents 
generally felt that in each case the questions examined had been appropriate to the. 
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point in the life of the program vton * coval^usstion had taken place. The point 
in the program's history when the eifs^^- ' ikm cDczurved was inot a factor in ^\plain- 
ing utilization in our data. TSere ^ »^ -i/st^^^^ttic relationship between his 
factor and degree or evaluation utilijaiti^^n* 

A factor that did emerge as somewha - \T1\poiT^^'::xit was the presence or ab^sence 
of other studies on the same issue. Stiu^^os tl:*»Jt broke new ground were paxticularly 
helpful because their potential for xedinc g . rtainty was greater. Nevertheless, 
such studies were viewed with some cautitf ^^^A^ns^ our decision-makers clearly 
favored the accumulation of as much infch i^^fh fisom as many sources as po^isible. 
Thus, those studies which could be relar^ ^Shy^f^ltsx studijes had a clear cumulative 
impact. On the whole, however, studies t ^.r l^rtsife new ground appeared to have 
somewhat greater identifiable impact. Al A^^If of the studies in our sample 
were' of this latter type. 

Finally, we asked our respondents about evaluator-government interactions. 
The studies in our sample appeared to have be^n based :on considerable interaction. 
Interactions were almost universally descxiied ;as cooperative, helpful, and 
frequent. Many respondents could offer hoari^OT stories about poor interactions on 
othqr studies, but with regard to the speci£in: :Study on liftiich we were conducting 
follow-up, there appeared to be few problrens;^. L^lrost evaluartors and project officers 
: reported that they had interacted regulara^^., Stoeover, they rated their interactions 
around these studies as at least average and often above average compared to their 
other evaluation experiences. There was no indication that utilization would have 
been, increased by greater government - evaluartonr initeraction than that which actually 
occurred, though the degree' of interacti^ whirh did occur was considered quite 
important. 

Overview of Findings on Factors 

We have reviewed eleven factors frequently identified in the' evaluation lit- 
erature as affecting utilization. Lateness of completion was not a problem in our 
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sajnple partly because of the nature of the impacts we indcntified* Methodologic i 
qisality did not emerge as an issue ±n the utilizatSion of the studies we examinee 
and respondents generally felt thait the methodology employed was appropriate to :h€e 
evaluation questions asked. Resources were not a problem^ The content of final 
reports, i.e., negative or positi^^ findings, was not predictive of utilization. 
Ifijtst of our sample evaluations looifced at central program objectives and revealed 
f^iew surprises about how programs were operating. The questions asked in these 
studies were generally appropriate to the point in the life of the program when 
tihe evaluation took place; and this factor too was unrelated to utilization in aissr}' 
^stematic way. Studies that broke new ground appeared to have somewhat greater 
impact in r-educing uncertainty in decision-making though the cumulative effect of* 
ery'aluations that related to other studies emerged as important. Interaction be- 
t-ween government administrators and evaluators was generally positive and occurred 
with marked frequency. ' ^ 

Only the importance of political considerations emerged clearly as a major factor 
in the utilization process. Our study confirms the conventional wisdom that eval- 
■uation research can be highly political in nature--and in use. But our respondents 
felt that politics could work to either increase or decrease utilization depending 
upon particular circumstances. Political considerations are a normal and probably 
inevitable factor in the use of any information that can reduce uncertainty and 
thereby affect power relationships wrtSSn and between organizations. By being aware 
of political consider^iitions evaluators^ can reduce their own uncertainty about the 
utilization of their iwork. 

None of those factors, at least .as we were able to explore them, helped us \ 
a great deal in explaining variations- in utilization. This is partly because, as 
noted .earlier, we had set out with a different concept of utilization than that 
which emerged from the data. Nevertheless, there was one major factor that did 
consistently arise in the comments of decision-makers, evaluators, and project officers 
a factor so crucial that respondents repeatedly pointed to it as the 
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Single most important el«flent 'in the mtilization proc^v 

What we are about to liiscuss will come as no surpr se to cotrr generous 
respondents and informanls. Ygir, we ddd not antlcipatn^. the xmortBrnze of this 
factor. Perheqps we can fault rtiJ^ srCmctural emphasis in much of sociology; or 
perhaps the problem lies: in th^^evaflisstion litjeratnire, in the ^ffidbSeation of 
:ra^±cinality and objectivity as itiie liniks between research finifiSngs ^and their 
latiaSization. Whatever the source of our initial narrow vision:;, we lieiieve 
tdhat these findings have profound implications fox: evaJLuative SKSearch and its 
utilization. 

What then is this factor -Irhat has emerged with such : striking clarity? 
Tbr lack of a better term, we have called it simply the personal factor. It is 
made up of equal partrs of leadership, interest, enthusiasm, determination, commit- 
ment, aggressiveness, and caring. Where tSie personal factor eiDEerges, evaluations 
Jia3Ke an impact; whece it is afeent thesre is a marked absence a£ impact. 

Social scientisjcs do not generally feel very comfortable with such personal 
JEactors. They smack too much of the great person theory of history. And so as 
fsoon as the personal factor emerges the social scientist turns away, presumably 
to search for the underlying structural: conditions that give xissa- to this irritating: 



personal phenomenon. And perhaps eventiually suc^ underlying con^'tions will be fully 

indentified and operationalized- But S)r the moinient we want to Hook directly at 

the factor itself--the effect of indi^zidual peqpiiie in a system where individuals 
■ • ■ _ ■ ■ . ■ i ■ ■ ■ 

are supposed to be interchangeable in Drganizastf&niaal zsoles and positions, bat where 

; ' they aren't interchangeable at ail 1- 

The personal factor emergasffiL nti^tt dramatically in inteneiews when:,, rhaviii® 

■ asked respondents to comment 0Tx:-1:he importance of eacih xof" our elszien utilizatiOE^^^ 
factors, we asked them to. idemtdrfy the single factor trfet was nrost important in. 
explaining the impact or lack of impact of that particular study. Time after time 
the factor they identified was not on our list. Rather, they xesponded in terms 
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©if the importance of incar^/iduall people, 

I woE^ld rank the ns^st iutpoxtant factor tills division dlTrector^s 
interest, [his] ini-^rest ii:k^smMBt±on, Not all; managers are that 
mo tivat:ed toward emluatium. (iBaK53::17) 

[T1ie:;::tsingle xmst ;iiigpGnctaiit ^ctor that had t&© gjreatest effect 
on how thass study gcs^ :usedi^^bBas] tissa principal investigator ... If" I ■ 
have to ipsSck a single :factisir, ZI^IJI pick people anf m 

Tliat .it came from the J1^3ica^ of the DirectoF-^^at ' s the most 
important -factor . • .T!he pxcmn^alL icame from the orfas^a of the :Dir- 
ector. Ilrhad had his atrteeaBcradin;^^^^^ was inters±id in it, and he 
implement:ieai: many of: the thiagpu ;QDM312:::213 

[Thef£Single mosit impuiiUuit ::Eac t or was that] tS^. people at the 
same levsi of decision-makimg -in- [the n^w office] SfiaKre Tiot interested 
in making decisions of the idnd^at tire people 

were, I thank that probably had gneatest impact;::, 'The fact that 
there was no one at [the new office^ aEter the transfer who was making 
programmatic decisions. (EK361:-27) 

Well, I think the answer tiiei^ is dn the quali-ties of the people 
for whom it was made. That's sort of :a trite answer, but it's true. 
That's the single most importantzSactDr in any study-^now that's utilized. 
(EV.232.:.22) 

Probably the single factor Tthat had tSse greatest effect on how 
it was used was the insas^HEsnce q£ the jggrsan repsomsible for the ini- 
tiating the study that tiifi&DirectrDr o£ b ecome familiar with 

its findings and arriverisf:^ a judpirent ^sir 'it:. (DM3:69:25) , 

[The most:: importaist factor was] real :i33!srolvement of the top 
decision-makers in the caaiceptualizaa i iagm and i^j^^n of the study, and 
theirrcommitment to tha^ ssmdy . CDM2fiK3f) 



Whilerthese comments Gtmceim tihe impoinsancse^ iinterested and. committed; 
individuals? in studies tMt'w^me: fflcsually used., saidiass that were not used->stand 
but in that there was cSten a*© cIctdt absrenoES oE -th^ per sima l iEactor. One evaluator., 
who was not: sure abmd: how ]fe£s snEi2% was raised suspected, it had not been used, 

■ remarked: "^I think -^±hat since^ itsaeicein, ent iwastrSrlt^rriblsr interested and the 

^ ■ w acad sa^icB. we-Awrasn 't interested in doing 

•it from a research point: of v:iew/ , nob ot^ was inter S32d." (EV264::14) 

Another evaluator was partaaiiiiaarly adamant and .artiojlate on the theory that 
i the major affecting utilrzat ion is the personal ^energy, interests, abiJIitiies, 
: and contact s of specific imdiivdduail s . TJiiis per son &ad ihad t hir ty- f ive yeaxs ^experience 
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in goy eminent, twent?^* of those years directly involved in research and evaluation. 

Hfi had -also ws»rked^^r several years as a private evaluasrion research contractox 

during which xime he had been involved in evaluating some- eighty projects for HEW. 

lihroughout his- respucmses to our questions on the importssce: of various specific 

factors in a££ecting utilirat:ion he returned to the thoiie. c^" individual suctions ♦ 

IVhen asked to identify the one factor tSiat is most <4Snporitainri;in whether a study 

gets used he summarized his viewgxoiiait: 

The most laniportant factor is desixe on t^he paist of the managers:, 
both the central federal msoia^ers and the site manecgfirs. I don't think 
there's [any doubt] , you..lGEn3»^. thait evaluation shouiEd: be responsive to 
their needs, :and if they lisEULe a rsal desire to get coni with whantevesr it 
is they're suig)os:ed to do, trkey'n* apply it. And iS 3ghe evaluatlfinrs don 't 
meet their needs they won't. About as simple as yoaixan get it^ T think 
the 1 whole process is fax mc3Te dependent on the skinEssrof the pTOpXe who 
use it than it is on^ the sc)Xt of peripheral 

Institutions are tough as heJil to change. You^ caii?tt:xhange .an :Sis 

by coming and^^doiing an evaluation with a hal:o,. InssgitutTDTis arg changed: 

by people, in tims?, with a constant: pluggin^.away ast "the purpose you 

want to aocomplish. And if you don't watchamit, it: ;s:lid^ back^ ;CEV346:iI5-16) 

His view had emerged early in the interview ^en he cSisscribed how ^valuaitions 
VF^^e used in OEO. 

In OEO it depended on whD the .program^^rfftcer ^4tfas, on the 
program iteview of ficials, on iprpgxam monitois fm* each of these grant 
programs: Where they were^ggi^sive prograiitt fKople, they used 
en/aiuations whether they lundejrstooxi them orldPfe. TShey the^^^ 
""^ai^iSvemSi^^ 

nations why the records ^\^ieasenik^t la^ais s^vmy tSieriasfports weren ' t 
complete cir whatever . Wheare it±.ey, viS^^ prqgrmir; o£Eiciais in OEO- 
were una^i:te.ssive, passive — ^mxtniirng:! ! 

Same thing Vs true at the ::proj\ec±: ievei- Wh^ene yoa^haad^a gp^^ 

who was ^aggressive and understmd ^isrhat ilihejThsll tfas^ 

and he us:ed::it as leveragerto ccffian^ what went on within his program. 

Those who weren't--nothing. (EAB4ftd5:) 

Nor did !*h:e view this emphasis" on tfe indalvidual as^^ i^^ evaluation was 

rsOTply a political tool". When aslsed bo*^ politicaa considerations affected eva- 
Ixations, he:;:^^ 

I don't think it's poliEiritra^ . :a3i laHI.. Oh, thsEsis .sotob prjessures 
every once in a while to ti^::snd jger -tnoxe efficie^ 
but es:^, but I don't think thatr'is theTsa in course. ^ 1^ 
:3how. thee administrators of taBS::prograin viev'- thanseiHv^es, -their respon- 
sibilities. That's the contro lIimg:;fEacl:r^r;:; I:.don''t:: tinnk it 's political 

^ ' in iany way, (EV346:8) 
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Later he conunented: 

It always £all5 back to the view of the administrator sm£ tiis 
view of where his perogatives are, his responsibilities. A good 
manager can manage with or without evaluations and a poor one caam^t, 
with or without evaluations. It just gives him ssane insights .rjntp • 
" ' ttfhatt he should or shouldn't be. doing, if he's a good manager. ET* 
they're poor managers, well . . . (EV346: 1.1) 

On his comments about each possible factor the same theme emeiseii. 
Methodological quality, positive or negative findiiiigjs, tlie degree tci«&ich the 
findings were expected — he always eventually retnraesi to the themes cfff matcagerial 
ihtei:est, competence, confidence. 

Ihe good manager is aggressive, open, confidenrtt:, anxious to intesrchange 
ideas. He's not defensive. Rather, ''he's interesited in finding out «ar. your 
views are, not defending his.... You Know my sample is relatively smali^ but IM 
say propably there are a quarter (25%) of what I'd call good ma:»ig^rs- . 
(EV34 6 : 1 5 ) These, he beJiieves, are the people who us:^ ea^lgaatiim resga ^srix. 

Our sample includes .another rather afemant rarticuliaticTi o£"^t:his ■premise. 
An evaluation of a pilot program involving: four majcir pisr©j£sits twas nmdertai 
at tte instigation of the program administtncat33r-. He JDS&ifce a. spec±aiIL a^ort to 
make sure that his question (Were the pilot: pro jiects ccsg^le oSnbeias^ ^xtenieaa. 
and generalised?) was answered. He guaranteed this l^sonally itsa:ldii^ 
active interest in all parts of the study... The adminisrttrator had. ^bo&eiit lEavorabl.e 
to the program in principle, was uncertain what th*^ icest^lts wotilidi :be>,, bcit was 
''hoping the results would be positive.'^ The -ev^isa^dfflL 122:01^ IK&e 

administrator was ^'surprised, but not alarmingly ssc:^ Wfir haa (expense a nmor^Tpossiitiace:^. 

finding or we would not have engaged in the poMot Htudices . (DlSBK7/:Ii35 Therprogt- 
roram was thus end^d with the evaluation carryingc'^sfemt a. th±rd. ^af the wreight of 
tare total decision.:'' (DM367:8) 

The evaluat or interview on this case complefceay^ substantiated titte administctatm-^^^^^^ 
aiescri-ption. The findings were^ specific and cleacr^ The progr.aim' "^v«as?:::not^^i^^ 

• • 3.6 



And thus the evaluation had a substantiajL , direct impact on that decision. The 

question then becomes why t±is study had such significant utilization. The 

answer from the decision-manicer was brief and to the point: v 

Well, [the evaluECtion had an impact] because we designed the 
project with an evaluartion component in it, so we were expected to 
use it and we did Mot just the fact that [evaluation] was built 
in,/but the fact that jte built it in on purpose. This is;, the agency , 
head and mvself had broad responsibilities for this, wanted the eval - ' 
uation study results and.we expected to use them; Therefore they were 
used. That '5 my point . If someone else had built^it in because they 
thought it was needed, and we didn't care, I'm sure the use of* the 
study results would have been different. (DM367:l2) 

As the decision-maker tells the story it had taken a great deal of direct 

interaction to be sure that the right question was evaluated. . 

"The initial dssigiB stages went round and round because they 
[the evaluators] kef^t trying to answer a different question than 
the one we wanted answerred ...If we had dropped it with them right 
then and said go ahead and do your own thing with it, it would 
not have been useful, .^il have a feeling I'm becoming redundant. 
The greatest single: facror [explaining utilization] was that the 
question we wanted ansv^red was the question they did at least try 
to answer in the study. (DM367:l6) 

The evaluator, (aniexteamal agent 'selected through an open RFP process) , 
completely agreed that;: i J 

" The principle reason [for utilization] was because :the> decision- 
maker was the guy who requested the evaluation and who- used : its; results. 
"^ That is , the orgahiTa?tT5na;l"'lii^^ t'he^61il:)^ihayer^^^ 

evaluator was almost zero in this instance . 'That 's, the most: important 
■ ' reason it had an impact ;*' (EV367 : ^ . ^ 

Well, I guess the point is that ,the project was really monitored 
by the decision-makor rather than the project officer ... It was the fact . 
^hat the guy vho was asking the .question was the guy^. who was going'^to 
m ake use of the -answer . So" it might be interesting , to experiment w 
^ - of having the guy whp; needs the the answer to the que^^^^ 

: " run the contract. Might be interesting. Might be; chaos .^^^^^/^^^^ 'v^^^^ 

H-ir: ,. ; An initial problem arose because the project officer had vyritten the RFP. 

:Sa^: the evaluator recalled the situatiori, the RFP was highly mislea^irh^ 

' ' If I had done exactly what the RFP asked for and turned in a report . 

: * • - . that was responsive to the RFP but not to what I was very qle^^^^ 

' i ■ the kinds of questions; they wanted answered, they would have gotten a 

^ . differeiit report". As a matter of fact, let me just tell youfthe essence 

: ; . : of ithe 't I had almost , no direction from the government ,ras I 've sr^ii^ 

. except that the gay kept saying, well here on point 8, you' ve got; to do 8 : 

fe ; on the contract. 37 



; . : So when I turned in the draft" o£ the report, I did points 

1 through 9 and put that in the final report. Then I essentially 

; w^^ after that and inade that the last-half o£ the 

report. It was a detailed description of the activities of the 
program, it came to very specific kinds of conclusions. It wasn't ^ 

■ ::v what was asked for in the RFP, but it was what they needed to 

: answer the question in the RFP. [The decisioh--maker] read it and 
; the comment back was, "It !s a good report, except for all that 
crap^ in. the .front." ' 

Okay, so I turned it around then in the final draft 
■ : . ■ alli t^ front into an appendix. And 1^^^^ 

■ - • the report, it has a big, several appendices . All of that , :if you 

compare that carefully to the contract , all that crap in the : ■ ; 
: appendix is what ^,1 was asked to get .- All the stuff that ' coristi^ / ' • 

the. body of ^he report was above and beyond the call; (EV367:12) : ^ ^ 

What emerges here is a picture of a decision-maker who knew what^i 

he wanted, an evaluator committed to answering the decision-makerVs question, and^^- 

a decision-maker committed to using that information. The result was a high level 

of utilization in making a decision contrary to the decision-maker' s initial 

personal hopes. And in the words of the „evaluator, the major factor explaining 

utilization was ' 

"that the guy who's going to be making the decision is aware 
of and interested in the findings of the study and has some hand 
in designing the questions to be answered, that's a very important* 
point." ; (EV367:20) : . 

The decision-maker's conclusion is so similar that it sounds like collusion; 

•^v Evaluation research : WellJ. guess I would_affirm that in 

many cases it has no impact for many of the reasons that the liter- 
. ature has suggested. But if T were to pick out factors that made a — - 

positive contribution to its use, one would be that the decision- 
makers themselves wanted the evaluation study results . I've? said 
that several times . -If that is not present, it is hot surj^rising 
that the results aren ' t used. (DM267:17) : : 

This point was made oif ten in the interviews. One highly placed and highly ^ 
experienced administrator offered the following advice at the end of a four hour , 
•iinterview:- ; \, • 



Win over the program people. Ma^ sure you're hooked into the 
person who's going to make the- decision in six months from the ' 
time you're doing the study, and make sure that he feels it's his 
study, that these arfe his ideas, and that it's focused on his ;:/;;|; 
values • ...I 'm sure it enters into personality things. ... (pM283:40) , 



. The personal factor applied not just to utilization but to the whole 
evaluation process. Several of the studies in our sample were initiated completely 
by a single person because of his personal interests and information needs. One ^ 
study in particular stands out because it was initiated by a new office director 
•with no support internally and considerable opposition from other affected 
agencies. The director found an interested and committed evaluator. The two 
worked closely together. The findings were initially ignored because there was 
no political heat at the time, but over the ensuing four years the director and 
evaluator worked personally to get the attention of key Congressmen. They were 
finally .successful in using personal contacts. The evaluation contributed to the 
eventual passing of significant legislation in a new area of federal control. 
From beginning to end the story was one of personal human effort to get evaluation 
results used. 

The specifics vary from study to study but the pattern is markedly clear: 
Where the personal human factor emerges, where some individual takes direct / 
personal responsibility for getting the ini^ or mation to the right people, evaluations 
have an impact. Where the personal factor is absent^there is a marked absence of 
;, impact .^-^-Utilization is not simply detearmined by some configuration of abstract 
factors; it is determined in large part by real, live, caring human beings . . 

Implications of the Personal Factor , in Evaluation 

If, indeed, utilization is to a large extent dependent upon the interests, 
capabilities, and initiative of individuals, then there are some profound impli- 
cations for 'evaluators. First, evaluators who care about seeing their results 
■•utilized mu3t take more seriously their responsibility for identifying relevant 
decision-makers. Relevancy in the context of the personal factor means finding 
decision-makers who have a genuine interest in evaluation information persons 
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who know what questions they want answered and whis) know how they can use 
evaluation information once findings acre available. Such individuals are 
willing to take the time and effort to iiirteract wx£h evaluators about their 
information needs and interests. 

Secondly, formal position and authority areoniy partial guides in identifying 
relevajit decision-makers. Evaluators must: Sud a strategically located person 
. (or persons) who is enthusiastic, commrtted, xompetent , interested, and aggressive. 
Our data suggest thatnnore :may be accomplis[^j©d b>^ working with a lower level 
person displaying these characteristics than in working with a passive, disinterested 7 
person in a higher position. f-v: 

Third, regardless of what an RFP caiEls jEbr, the most valuable information ' 
with the highest potential for u t i 1 i z Btfmm is tfca t information that directly 
answers the questions of the individuallCs) idEortrfied as the relevant decision- 
maker (s) . Requests for ^jroposals (RFEs> niay be written by individuals other than 
the decision-makers who really need and waaiit the evaluation information. It_jDehooves 
evaluators to clarify the degree to which an RFP fully reflects the information needs ^ 

of interested government officials. ^ . i-; 

Fourth, attenticfn to the personal :i&c£i3r rmay assist not only evaluators in 
their efforts to increase the utilx2al:ion of their research, but attention to 
the personal factor can also aid decision-makers in their effort to find 
/ evaluators who will provide them wirth relevant arid..useful information. Evaluators j.? 
;! . who are interested in and knowledgeable about what they're doing, and evaluators 

who are committed to seeing their findings utilized in answering decision-maker's : 11' 

questions will provide the most useful Information to decision-makers. -vij 
■ i- V ■ Fi there are political implications for both evaluators and decision- . -:yf 
/^ makers in explicitly recqgni zing and acting oh' the importance of the personal ':M 
; factor. To do so is also to accept the assumption that decision-making in - 
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government is likely to continue to be a largely personal and political process 
teS i*ather-:thari; a rationalized and scientific process. This assumption means that 
i^;!; rli3ither the decision-m nor the evaluator is merely a technician at any stage 
iyg iin :;the evaluation process . The personal factor is important from initiation 
VI vo;f , the study thro data collection stages as well as in the final • 

-V -report have shown 

•little interest in the study in its learlier stages our data suggest that they are 
not likely to suddenly show an interest in using the findings at the end of the 
study. Utilization considerations are important throughout a study not just at 
the stage where study findings are disseminated. 

Finally, the importance of the personal factor suggests that one of the 
major contributing reasons for under-utilization of evaluation researches the 

high degree of . .instability in Federal program operations . This, instability, 

based on our data, is of three kinds: (1) high turnover rates among senior 
government staff so that the person initially interested in an evaluation may 
be in an eritTrely different office before the study is completed] (2) reorgan- 
ization of government offices so that decision-making patterns are unstable,, 
personnel are frequently rearranged, and responsibilities are almost constantly 
changing; and (3) program mobility as programs move from office to office 
: (e.g. , OEO to HEW) even if no formal, structural reorganization occurs. 

We found the instability of Federal organizational charts and the 
- mobii; ty/ turnover among staff Lo be substantial. In tryingT:©' retrace the history 
. of evaluations we frequently got a response like the following: *'I/ve had so 

many changes in organizational assignments since the^i, I don't remember." (EV201:6) 
V Asked about utilization of evaluation the same person responded: 

„ Wei 1 . since you !,re not going to identify, me and my name , 1/ 11 tell you 

what I really think, and that is, I think these plans go up to the 
planning office and the rotation of personnel up there in the Js 



of fice "is so fast und so furious, that they never get a chance to react 
to them, [It] just sits. We know that happens sometimes, because the 
guy who asked for it is gone by the time it gets, up there. (EV201:8) 

1 The problem of instability appears to be particularly critical in actually 

implementing recommended changes. 

It was easier to get recommendations through with senior, management ^i;; ■ , ■ 
approval . ' I mean, they read it and they could easily implement some of 
the . areas,^and: they7 ""I m^ean; at leaist they; could "impilem^ - 
anyhow. But still the problem, in any study or anything of; this calibery ■ / 
it's. up to' the people in the operating unit to make the. change .. And there's 
no way for senior management to measure that change. Ther^*i's no way to 
see that it was even done. You know it's the old thing, ur/ you know, 
they tell people to do things, but in areas that require technical expertise 
there 's no way to see that the change was done. And so people in the : , 
operating area many times would just wait out~ the person, yoU 'know,^ some 
of these people have been through 3 directors, 5 associate directors, 
. you know, and, they don't want to do something.: 

they knovf that if they sit long enough that that person will pass and 
someone else will come in with brand new ideas and, (DM312; 15) 

Another evaluator found the same problem in the field. "I have spoken , 

with project directors who tell me they really, have no one to talk to because 

at the regional level the project officers and plrogram officers change jobs so 

frequently that there's no continuity." CEV346:7) Evaluators commented that 

V'"* 

it was common experience to go through several project officers on an evaluation. 

Our own experience in trying to locate the respondents in the sample gave 
us a clear indication of this instability. Few of our interviewees were still 
in the same office at the time of the interview that they had been in at the time 
of the evaluation two to three years earlier. We still haven't been able to 
construct a meaningful organizational chart of HEW locating the various office 
changes and agency reorganizations we encountered. 

These structural conditions of mobility and instability make application 
of the personal factor in locating relevant decision-makers or evaluators a risky 
business. That key person you locate may be gone by the time the study is 
completed. Yet these same structural conditions of mobility and instability may' 
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welHfee the underlyinjg reasons why the decision-making process in the Federal 
government has been and continues to be a highly personal and political process . 

Conclusion 

Two major themes emerge from this study of the utilization of evaluation 
; research. First, we found that much of the evaluation literature has considerably 
overestimated the kind of impact evaluation research is likely to have. V 
Second, the importance of the personal factor in evaluation research, particularly 
. the utilization process, has been considerably underestimated. 

The two themes are directly linked. The impact of evaluation research is 
most often experienced as a reduction in the uncertainty faced by individual 
decision-makers as they attempt to deal with the complexity of programming 
reality. Evaluation information is one piece of data available to decision- 
makers. It must be assimilated and fitted into a contextual whole/ '^The results 
are never self-explanatory.*' (EV209:9) The translation, the interpretation, 
the meaning, the relev^-^nce- -these things are established through the interactions -'^>^^^^ 
over time of individuals who care enough to take the time to make the contextual 
fit, and then are interested enough to act on the basis of that contextual fit. 

It is an energy- consuming process. Energetic and interested people in 
government can and do use evaluation research, not for the making of grand 
decisions with immediate, concrete, and visible impacts, but in a more subtle, ' 
clarifying, reinforcing, and reorienting way. Evaluators, then, might do well 
to spend less time lamenting their lack of visible impact on major decisions and 
more of their time providing relevant information to those key persons of energy 
and vision whose thoughts and actions, to a substantial extent, determine th' 
general direction in the evolutionary process of program development. It is in 
consciously working with such decision-makers to answrr their questions that the 
utilization of evaluation research can be enhanced. 
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